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Direct assessments of adult 
skills and competencies 


Numeracy and literacy in the adult population are key 
skills that transform lives, promote social inclusion and 
contribute to sustainable societies. Without these skills, 
adults are at risk of disadvantage. Furthermore, innovation 
will not as easily translate into a growth in well-being, 
while individuals and countries will be less prosperous. 
Getting the best returns on investment in the development 
of adult skills and competencies requires comprehensive 
data to assess what people know, what they do with 
what they know, and the extent to which they are 
equipping themselves with the skills they need to sustain 
themselves in the twenty-first century. 


While collective efforts have led to continuous progress in 
increasing literacy rates, global data on literacy, based on 
conventional proxy indicators, is not sufficient to offer a 
comprehensive picture of the literacy landscape. To give an 
example, the global literacy rate for adults aged 15 years 
and older was 86% in 2018, a figure that is almost unchanged 
since 2016. However, in absolute terms, the number of 
non-literate youth and adults increased by 23 million to 
773 million between 2016 and 2018 (UNESCO, 2020) - 

a stark reminder of the task ahead if the international 
community is to deliver on its education promises by 2030. 


Currently, only 13% of countries have data on literacy 
and numeracy skills based on direct measurements 
(Subosa and West, 2018). Literacy estimates are mainly 
based on a self-reported measure of reading, which 
distinguishes between those who cannot read or write 
at all and those who have some skills, and usually does 
not measure numeracy competencies. In contrast, direct 
measures of literacy and numeracy determine how well 
individuals perform the key skills society needs and 
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produce a detailed analysis of the distribution of skills 
in the adult population. 


Moreover, direct measures of the skill levels of adults in 
literacy and numeracy help to: 


e determine the sub-groups that are not equipped with 
sufficient skills and are more likely to be left behind. 
For example, in Germany, the second Level One Survey, 
conducted in 2018, assessed the reading and writing 
skills of the German-speaking adult population and 
gathered detailed data about the distribution of three 
lower levels of literacy proficiency, taking into account 
a series of socio-economic variables (gender, age, mother 
tongue, educational qualification, among others) 
(Grotlschen et al., 2019). With this information, the 
government can implement targeted basic skills training 
opportunities for the most fragile sub-populations in 
appropriate settings, such as in the workplace; 


e examine the impact of skills on a range of economic 
and social outcomes. In countries participating in 
PIAAC,' the survey results are used to determine the 

influence of skills proficiencies on professional wages, 

ikelinood of employment and health (OECD, 2019); 


e evaluate the performance of education and adult 
training, work practices and social policies in developing 
the skills that are required to create and promote 
knowledge-based societies. Kenya, for instance, designed 
and carried out the Kenya National Adult Literacy 
Survey (KNALS) in 2006 to assist in the development of 
indicators used to assess progress on the implementation 
of education-related programmes (KNBS, 2007); 


‘Over 40 countries take part in the Organisation for Economic Co-operation and Development (OECD) Survey of Adult Skills, 
which is part of the organization's Programme for the International Assessment of Adult Competencies (PIAAC). 
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e identify the policy levers that can reduce inequalities 
and deficits in key competencies. In Bangladesh, data 
from the national Literacy Assessment Survey (LAS) 
is used to inform the country’s strategy to improve 
literacy of youth and adults through high-quality early 
education and lifelong learning (UNESCO, 2010). 


The challenge of conducting direct assessments 
in low- and middle-income countries 


Direct assessments produce data that policy-makers 
could use for policy development (UIS, 2018), but past 
experiences in some low middle income countries suggest 
that potential challenges could discourage them from 
investing in direct assessments (UIS, 2017). 


One of the primary challenges is cost (Guadalupe et al., 2009). 
Direct assessments are more expensive to conduct than 
usual household surveys. To identify accurately the lack of 
skills in certain sub-populations, for instance, data need to 
be collected widely, even in rural and remote areas. Data 
collection in fragile states is even more expensive because 
of the lack of basic infrastructure and security requirements. 
Moreover, for direct assessments, which are also designed 
to collect data on the literacy environment and individual 
literacy experience, the survey is longer than it is for usual 
household surveys and should be conducted by experienced 
facilitators, whose training adds to the running costs. 


Developing assessments and surveys that are comprehensive 
yet transparent is another challenge, particularly when 
producing test items that will be adapted to country 
contexts (Fransman, 2005). In some cultural environments, 
respondents may interpret and answer test questions 

in a way that could produce culturally biased results or 
counter-intuitive outcomes (Maddox et al., 2015). A strong 
infrastructure that meets generally accepted scientific 
standards is therefore needed to support and conduct 
data collection and analysis in the field (UIS, 2017). 
Broad-based tests that measure skills across cultures and 
social settings can be developed with appropriate 
investment and capacity-building efforts. 


A final challenge is in deciding on the number of languages 
in which to test the adult population (Heugh et al., 2017). 
nsome countries, in addition to several national languages, 
many local languages are spoken. Assessing skills in all these 
anguages adds to the complexity and the cost of the 
survey. Adequate sampling design is needed to establish 
stable statistical properties for each tested language and 
to minimize administrative costs by targeting populations 
who speak in and use the language of assessment. Ad- 
ministering a multilingual survey also requires multilingual 
interviewers and a questionnaire design that is easily 
adaptable to the language spoken and used by household 
members (Koch, 2018). 


Policy recommendations 


Accurate data on literacy and numeracy is key for a country 
to guide its policies and programmes to monitor progress 
towards national targets. Direct assessment is the only 
way to obtain a full skill proficiency distribution of the 
adult population, to track the progress of sub-groups over 
time, and to provide sufficient information to develop 
specific policies. Based on international experience and 
the latest research findings, the following recommendations 
are offered to guide policy-makers in implementing direct 
assessments in lieu of proxy surveys. 


Integrate direct assessment surveys of adult skills 
in relevant national strategies and policies: 


1. Set up a technical taskforce to advocate for national- 
level direct assessments of adult skills and competencies 
and to lobby relevant national and international 
agencies for technical and financial support. These 
direct assessments should then be used to develop a 
national literacy strategy that operates across sectors, 
inclusive of the higher education and vocational 
sectors, to inform policy-making, including for the 
setting up of formal and non-formal education centres 
and the creation of adult literacy programmes. 


2. Prepare an implementation plan and budget to raise 
funds. If the national literacy budget does not allow 
the conducting of national-level direct assessments, 
a proposal with an estimated budget needs to be 
developed to raise funds from donors. 


Evaluate existing national infrastructures and 
capacities for conducting direct assessments, and, 
if necessary, build and strengthen them: 


1. Set up an in-country expert committee to develop 
an implementation plan and budget. Given the 
complexity of national-level direct assessments, the 
committee should comprise literacy and assessment 
experts and data users. Also consider assembling 
a cross-sectoral team built on a lifelong learning 
approach to integrate direct-assessment outcomes 
from early childhood to adulthood in national policy 
development. Good communication among 
committee members is essential to ensure success in 
the generation of relevant, quality and reliable data 
on adult skills. Once the committee is formed and the 
terms of reference agreed, some initial steps would 
include (UIS, 2009): 


e Define the policy questions the survey should 
answer and identify population groups that are of 
most interest, such as rural youth, women and 
at-risk youth in urban areas. 
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e Decide on the language of assessment. Be aware 
that the decision to assess one or multiple languages 
has political, financial and operational implications. 


e Identify potential (practical) issues and/or potential 
gaps in implementation and develop appropriate 
strategies. Prepare alternatives for worst-case 
scenarios with rigorous quality assurance to ensure 
data quality. 


e Identify potential partners in field implementation. 
If household surveys have already been conducted 
in the country, negotiate with the implementing 
organization to incorporate direct assessments. 


2. Identify existing direct assessments and determine 
testing capacities and experience. If capacity in 
conducting national household surveys and school- 
based assessments is available, identify which agencies 
and technical experts to involve in the implementation 
of adult direct assessments. If necessary, review the 
available technical documents of existing cross- 
national programmes or study success stories from 
countries that have implemented adult literacy 
surveys in the region to draw lessons. 


3. If the country has no experience in direct assessments, 
organize a team of technical experts and provide 
them with relevant training in adult skills assessment. 
In addition, consider participating in cross-national 
assessment programmes, which could offer the use of 
existing instruments and support from international 
partners. For example, in 2003, Mongolia joined 
the UNESCO Institute for Statistics’ (UIS) Literacy 
Assessment and Monitoring Programme (LAMP), which 
was developed to gather quality data on literacy 
through new national household surveys. Based on 
its experience, the country was able to provide 
support to Viet Nam in training its own LAMP team 
and fielding the survey (UIS, 2009, p. 26). 


Ensure data are comprehensive and of high quality: 


In order to be suitable for use in policy development, data 
need to be accurate and easy to understand. To collect 
such robust data, the development of the assessment 
survey should be divided into three stages (UIS, 2019): 


1. Conceptual framework 


The conceptual framework is a theoretical framework 
that describes the cognitive process in learning a skill and 
helps direct the development of test items. For countries 
with a national adult literacy programme, the national 
literacy curriculum could be used as a basis for the con- 
ceptual framework. If a country does not have a national 


adult literacy programme, the conceptual framework from 
cross-national programmes can be used as a reference. 


2. Implementation framework 


An implementation framework is a set of processes to help 
collect and analyse data in direct assessments. It covers 
item development, population sampling, administration, 
quality assurance and data analyses. Use national expertise 
in household surveys and school-based assessments to 
develop instruments, guidelines and data collection plans 
to administer adult direct skills assessment. Given that 
quality assurance and analysis, especially psychometric 
analysis, is different from household surveys, seek 
inspiration from cross-national programmes. 


3. Reporting framework 


The reporting framework helps organize the gathered 
data and put it into easy-to-understand performance 
descriptors, i.e. ‘basic’ ‘proficient’ or‘advanced’ The 
performance descriptors help data users understand 
what those at a certain performance level can do. 

As developing performance descriptions is new to many, 
seek support from cross-national assessment programmes 
and the assessments community. 


Use the data from direct assessments to make 
changes locally: 


1. Given the basic human right to education and its 
implications for society, the outcomes of direct assess- 
ments should be used to identify the emerging demand 
for foundational skills to contribute to persona 
enrichment, sustainable economic development and 
improved civic participation. In this regard, develop 
relevant policies and set up new learning opportunities 
to address learning and skill needs and the demands 
of those groups that are left behind. 


2. Make effective use of data to improve adult literacy 
policies and programmes. Use the outcomes of direct 
assessments to identify sub-populations that are most 
at risk and the type of skills that are demanded. With 
proper use of direct-assessment outcomes in the national 
literacy strategy, the data from literacy surveys could be 
used to develop or improve the teaching and learning 
modules in formal or non-formal adult literacy programmes. 


3. Ifa country’s direct assessments are well designed and 
executed, the results can be used for monitoring and 
reporting on Sustainable Development Goal (SDG) 
indicator 4.6.1, which measures the ‘proportion of 
population in a given age group achieving at least a 
fixed level of proficiency in functional (a) literacy and 
(b) numeracy skills, by sex’ (UN, 2018). 
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